Notes on the Cramér-Rao Inequality

Suppose X is a random variable with pdf fX(x; θ), θ being an unknown parameter. Let X1, . . . , Xn be a random sample and θ̂ = θ̂(X1, . . . , Xn). We’ve seen that E(θ̂), or rather E(θ̂) − θ, is a measure of how biased θ̂ is. We’ve also seen that V ar(θ̂) provides a measure of efficiency, i.e., the smaller the variance of θ̂, the more likely E(θ̂) will provide an accurate estimate of θ. Given a specific unbiased estimator θ̂, how do we know if it is the best (most efficient, i.e., smallest variance) one, or if there is a better one? A key tool in understanding this question is a theoretical lower bound on how small V ar(θ̂) can be. This is the Cramér-Rao Inequality. From now on, we assume X is continuous and θ is a single real parameter (i.e., there is only one unknown). We will also assume the range of X does not depend on θ. To be more precise, we will assume there exist a, b ∈ R ∪ {±∞} independent of θ such that { fX(x; θ) > 0 if a < x < b fX(x; θ) = 0 if x < a or x > b.